The Refraction API | Inputs and Outputs

You can use the refraction API to fix tool calls. As an input, you need to provide, at a minimum, the tool call and the backing tool specs (catalog).

from refraction import refract
result = refract(<TOOL CALLS>, <CATALOG>)

2.1 Accepted Inputs

The form of acceptable tool calls and catalogs follow a wider range of formats, including the NESTFUL form and common tool calling patterns using for calling LLMs. In the following we discuss the raw forms of inputs and outputs. In the next chapter, we will discuss how to use the refraction decorator to annotate tools directly.

2.1.1 On a single call using the NESTFUL SequenceStep Object

from nestful import SequenceStep
from refraction import refract

call = {
    "name": "Tripadvisor_Get_Restaurant_Details",
    "arguments": {"restaurantsId": "$var2.restaurantsId$"},
    "label": "var1",
}

result = refract(
    sequence=SequenceStep(**call),
    catalog=catalog,
)

The catalog input follows the NESTFUL schema. Check here for an example of what the schema looks like. You can fetch a catalog like so:

from nestful.data_handlers import (
    DataID,
    get_nestful_catalog,
    get_nestful_data,
)

catalog = get_nestful_catalog(name=DataID.GLAIVE, executable=False)
sequences, catalog = get_nestful_data(name=DataID.COMPLEXFUNCBENCH)

2.1.2 On a sequence of calls using the NESTFUL SequencingData Object

You can refract on an entire sequence together (this is extra helpful because you can validate the flow of data across multiple tool calls).

from nestful import SequencingData
from refraction import refract

calls = {
    "input": "Find flights from New York to London that ...",
    "output": [
        {
            "name": "TripadvisorSearchLocation",
            "arguments": {"query": "London"},
            "label": "var1",
        },
        {
            "name": "TripadvisorSearchHotels",
            "arguments": {
                "geoId": "$var2.ID$",
                "checkIn": "2024-08-15",
                "checkOut": "2024-08-18",
            },
            "label": "var2",
        }
    ]
}

result = refract(
    sequence=SequencingData(**calls),
    catalog=catalog,
)

If you do not want to consider other APIs in the catalog mentioned in the call(s) while coming up with a fix, then use the following input. This will make validation considerably faster.

refract(..., use_given_operators_only=True)

You can also provide a timeout like so:

refract(..., timeout=5)

2.1.3 Using a Formatted String Form

You can also use the prettified string form as input directly.

from refraction import refract

result = refract(
    sequence=[
        'var1 = TripadvisorSearchLocation(query="London")',
        'var2 = TripadvisorSearchHotels(geoId="$var1.ID$", checkIn="2024-08-15", checkOut="2024-08-18")',
    ],
    catalog=catalog,
)

2.1.4 Using OpenAI / LangChain tool calling formats

You can also use the format of tool calling used in the OpenAI and LangChain tool calling APIs. These are very commonly used in LLM prompts. Check here for an example.

Check here for instructions on refraction while calling tools directly.

Using a mapper

You can provide a list of mappings (between output and input fields of APIs in the catalog) to help the refractor compute better fixes when the tool calls are corrupted. Mappings should be precomputed and passed in as inputs at runtume.

You can compute a list of mappings from a catalog like this (the threshold parameter determines how strong a mapping needs to be at a minimum, and the top_k parameter indicates how many maps for the same field to keep around):

from nestful.data_handlers import get_nestful_data
from refraction.mappings.compute_maps import Mapper

sequence, catalog = get_nestful_data(executable=True)
mapper = Mapper()

computed_mappings = mapper.compute_maps(catalog, top_k=1, threshold=0.80)

Optionally, you can also cache (and merge with existing) the mappings used in already existing sequences like this:

from refraction.mappings.utils import cache_maps, merge_maps

cached_mappings = cache_maps(sequence_data)
cached_mappings.extend(computed_mappings)

merged_mappings = merge_maps(cached_mappings)

You need to pass on the mappings into the call like this:

from refraction.schemas import Mapping
from refraction import refract

result = refract(
    sequence=...,
    catalog=...,
    mappings=[
         Mapping(source_name="skyId", target_name="destinationSkyId"),
    ],
)

Or using the refractor class, initialize this once. This will automatically compute potential maps between APIs in the catalog.

from refraction.integration import Refractor

refractor = Refractor(catalog=catalog)
refractor.initialize_maps()

refractor.refract(...)

👉 If you are integrating with a long running agent, and you are not validating a call in isolation, this is the recommended way to use the API.

Using a memory

You can also use a memory if you are executing a sequence but validating tool calls one step at a time. In the sequence example above, the memory would look something like this after executing the first step:

from refraction import refract

memory = {
    "var1": {
        "geoId": "...",
        ...
    }
}

result = refract(
    sequence=['var2 = TripadvisorSearchHotels(geoId="$var1.ID$", checkIn="2024-08-15", checkOut="2024-08-18")'],
    catalog=catalog,
    memory=memory,
)

👀 Note the unresolved reference in the actual call. If you want validation of data flow to be enforced by the refractor, then you must produce calls without yet resolving the references.

2.2 Output Format

The output of refraction is an object that contains a determination of whether the tool call is schematically correct or not, a corrected call if possible, etc. The output also contains a prettified diff string of what went wrong.

2.2.1 Result Object

Explore the result object to find out details about what happened during the refraction process. You can find out if it was successful by looking up:

result.report.determination

If this is True it means that the call is executable as is. If it is False, it is not; and you can look up the other parts of the result to see what went wrong.

If it is None, it means the refraction process failed for some reason, without a definitive determination (the result object will contain information about errors and timeouts).

Previously, we mentioned the need for guarantees. We also spoke about how the refractor does not look at the values assigned to parameters. This bears out here in the sense that if the refractor says NO then something is definitely wrong and there is no point trying to execute this call. If it says YES then the call is executable but might be wrong in the values assigned to items.

2.2.2 Prettified Form

In case the determination is False, you can print out the prettified form for debugging, feedback, or whatever other reason, by doing this:

print("\n".join(result.diff))

  var45 = TripadvisorSearchLocation(query="London")
- var5 = TripadvisorSearchHotels(geoId="$var4.ID$", checkIn="2024-08-15", checkOut="2024-08-18")
?                                              ^

+ var5 = TripadvisorSearchHotels(geoId="$var45.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")
?                                            + +++ ^

2.2.3 Corrected Call Generation

If the determination was False but is fixable from within what was provided, you can generate the new corrected call like so:

corrected_call = result.corrected_function_call(memory, catalog)

You can check if this is readily executable (e.g. does not require extra slot-fills for missing parameters) by checking this:

cfc.is_executable

2.2.4 Prompt Generation

Finally, in case you want to pass on the output of refraction to a help a traditional reflection component, you can do so as follows. This results in a two stage integration in an agentic flow, as illustrated above. In the first stage, if the refractor says YES, then you can chose to attempt an execution. If it says NO, and the corrected function call as described above is also not executable, then you can pass on the feedback to a traditional reflection component.

from refraction import refract, generate_prompt

prompt = generate_prompt(
    result,
    sequence,
    catalog,
    memory_objects=memory,
    prompt_type=PromptType.WITH_SUGGESTIONS,
)

Here is a sample generated prompt from the refraction result to pass on to an LLM.

Please fix the provided tool call based on the issues outlined.

<tool_call>[
{'name': 'TripadvisorSearchHotels', 'arguments': {'location': '$var4.geoId', 'checkIn': '2024-08-15', 'checkOut': '2024-08-18'}, 'label': 'var3'}
]</tool_call>

The following are the identified issues:
Each issue is accompanied by guidance on how to fix it.
Consider the guidance, along with the provided tool specs, and memory, to come up with the final fixed tool call.

- Parameter location is not a recognized parameter for the tool TripadvisorSearchHotels.
- Parameter geoId is a required parameter for TripadvisorSearchHotels, but it is missing.
- Possible fix: Get value of parameter geoId by calling TripadvisorSearchLocation.
- Possible fix: Call TripadvisorSearchLocation with parameters: query.
- Possible fix: Get value of query from the user input: Book a hotel in Boston.